feat: CRD value accessors (native modules + raw objects, src + chart)#99
Conversation
|
Using this here to automatically inject CRDs into evaluation for type checking/defaults, without maintaining static CRD nix files. |
|
I have considered adding resource options generation at evaluation time like this before but I was concerned about evaluation speed with such a setup. Have you noticed major slowdowns with this setup? I'm not opposed to this setup in general but I will need to some time to review your changes. Thanks for the effort in improving nixidy! :) |
|
Also, I've gotta ask, was any of this LLM generated? I don't have any policy that disallows LLM usage, it's just good to know when reviewing it. Also there's no need for a |
|
Yes, I had Opus 4.8 extract the pattern out of my config for upstreaming/sharing. Feel free to critique it mercilessly. :) |
|
Also, I'd like to take this opportunity to unify the parameter naming between the
Ideally, they should both still take a What do you think about that? |
|
Finally have some time to properly sit down with a computer and give richer feedback.
No; in practice these stay cached within the nix store and make it much easier for me to bump chart versions -- it's no longer a multi-commit juggling act. Adjacently I did try folding the IFDs on nixidy/kubenix down to just one call; but it actually wasn't any faster for me at the end of the day. That's why the version here is the re-expanded one. :) I've been running a variant of this pattern for ~3-4 months, but my folded-IFD version not being worth contributing is what delayed my upstreaming. I also explored conceptually minimizing IFD as much as possible / moving some eval to pure nix as an academic exercise; but without a reliable native/C-based Side note; it's actually been nice to programmatically manage these this way; in my personal argocd deployment I have a 'bootstrap' app that includes all global CRD resources + namespaces so there aren't any cross-app namespace race conditions on things like cilium firewall rules/etc.
I think it's a good idea; will apply it to all 6 interfaces. I'll add a deprecation warning with rename instructions so edit: for clarity; when I said 'manage all crd resources' I meant it in the context of 'atm machine' :D -- declared resources are still in the their apps; the custom resource definitions themselves are the only thing cloned. |
…ound-trip
`fromCRD` generates Nix *source*, writes a `.nix` file, and you `import` it
back — a serialize/deserialize round-trip plus an import-from-derivation. For
flake consumers that just want the resource options, that intermediate file
is pure overhead.
`fromCRDModule` returns the resource definitions as a module *value*
(`{ lib, options, config, ... }: { ... }`) instead. It drops straight into
`nixidy.applicationImports` (which already accepts `functionTo attrs`), so
there is no generated file and no import IFD — only the unavoidable Python
crd2jsonschema parse, now shared between both backends via the extracted
`crdSchema` helper.
- runtime.nix: the per-module boilerplate that generator.nix inlines as
source (submoduleOf, coerce-attrs-to-list-by-key, the str/coercedTo type
overrides, defaults handling, ...), extracted as real functions.
- module.nix: a value-emitting port of generator.nix's schema walk
(genDefinitions / mapType / genResourceOptions).
- equiv-test.nix + `nix run .#crdModuleTest`: asserts fromCRDModule renders
identically to fromCRD for a CRD that exercises submodules, list-by-key
coercion and the global ObjectMeta metadata ref.
The text generator (`fromCRD`) is unchanged — verified to produce
byte-identical output — and remains for the CLI "commit generated types"
workflow, which needs source files.
The fromCRDModule prototype duplicated generator.nix's schema walk
(genDefinitions / mapType / genResourceOptions) inside module.nix — two
hand-kept twins of the type/coercion branch logic, the part most likely to
drift (flagged in review).
Extract the walk into walk.nix, parameterized by a `backend` of emit
combinators (types, mkOption, submoduleOf, coerce-attrs-to-list-by-key, ...).
Two backends implement it:
- backend-text.nix -> Nix source fragments; generator.nix assembles them
into the committable standalone .nix file
- backend-value.nix -> live values via runtime.nix; module.nix assembles
them into a module value
generator.nix and module.nix are now thin assemblers over the one walk; the
branch logic exists in a single place (net -197 lines).
Verified: file generator output byte-identical (argocd store hash unchanged);
fromCRDModule still renders identically to fromCRD (nix run .#crdModuleTest);
moduleTests 42/42; crd2jsonschema 18/18; staticCheck clean.
Remaining (irreducible) duplication: the runtime helper block exists as live
functions in runtime.nix and as inlined source in generator.nix's template.
Committed standalone files can't reference nixidy internals, and live
types/functions can't be serialized back to source — so the two forms must
coexist. The walk (the real divergence risk) is no longer duplicated.
crdObjects { src, crds, kindFilter ? [ ] } returns the raw
CustomResourceDefinition objects from a set of CRD YAML files as values —
the objects counterpart to fromCRD's types. Deployment-agnostic: it hands
back the manifests and lets the caller decide what to do with them (e.g.
apply them to a cluster), rather than baking in an install mechanism.
Reuses fromChartCRD's isWanted filter shape; kindFilter mirrors fromCRD's
(empty = every CRD).
Add the chart counterparts of the src-based accessors:
- fromChartCRDModule → resource type module from a chart's CRDs
- crdObjectsFromChart → raw CustomResourceDefinition manifests from a chart
Both share an internal mkChartCRDsYaml (raw `helm template --include-crds`),
so calling both with identical args templates the chart once. The output is
raw helm YAML parsed directly downstream — no klib.fromHelm/yq-go
re-serialization, avoiding its number-coercion quirks.
kubeVersion is now a first-class, overridable argument across the chart
family (fromChartCRD too), defaulting to nixidy's nixpkgs version. Consumers
templating CRDs for a cluster on a different Kubernetes version can target it
explicitly — previously the version was hardcoded to nixidy's pkgs.
Completes the CRD API matrix: {fromCRD, fromCRDModule, crdObjects} × {src,
chart}. fromChartCRD is unchanged for existing callers (same default).
Expand the accessor test (renamed crdModuleTest -> crdAccessorTest) from fromCRD<->fromCRDModule equivalence to the full value-accessor surface: - fromChartCRDModule renders identically to fromCRDModule for the same CRD (a local helm chart shipping the CRD in crds/, copied verbatim). - crdObjects returns the raw CRD manifests; kindFilter narrows (hit + miss). - crdObjectsFromChart returns the same objects as crdObjects. - the chart accessors honor kubeVersion: a templated CRD embedding .Capabilities.KubeVersion renders v1.31 vs v1.40 observably differently. Seven checks; the derivation builds iff all pass.
Drop the hand-rolled helm-template derivation in favor of klib.buildHelmChart, per review. Materialize its single-file output as $out/crds.yaml via runCommand cp rather than linkFarm: crdObjects reads crds.yaml at eval time, and a linkFarm symlink would make that read follow into buildHelmChart's separate output (a store path absent from the readFile string's context) which pure eval forbids. A real file keeps the read in-context.
`crds` meant two different things: a list of CRD YAML files (src-based fromCRD/fromCRDModule/crdObjects) and a kind filter (chart-based fromChartCRD/fromChartCRDModule/crdObjectsFromChart). Split into two self-describing names: - src-based: crds -> crdFiles - chart-based: crds -> kindFilter (same role as fromCRD's kindFilter) Every function keeps `crds` as a deprecated alias (new arg wins; falling back to `crds` emits a lib.warn pointing at the replacement) via a shared `renamedArg` helper, so existing callers keep working. The crd2jsonschema JSON contract is unchanged (still keyed `crds`). Tests cover both new names and the deprecated alias (9/9).
2e35200 to
a5e6a28
Compare
…s clash Address review feedback: - renamedArg now warns (lib.warnIf) when both the deprecated `crds` and the new arg are supplied, instead of silently ignoring `crds`. - Add a crdAccessorTest check exercising the value backend's specialMapKeys branch (module-only arg, no file-backend counterpart to diff against): two name-less ports with distinct merge-key values stay separate; the empty default collapses them.
…r rename - typed_resources.md: lead the CRD section with the output-shape × source matrix; add subsections for fromCRDModule/fromChartCRDModule (module value, no generated file) and crdObjects/crdObjectsFromChart (raw manifests); document the kubeVersion chart argument; rename crds → crdFiles (src) / kindFilter (chart) in all examples with a deprecation note. - architecture.md: describe the shared walk + text/value backends and the six-accessor matrix; update the CRD example args. - llms.txt: mention the module-value and raw-object accessors.
|
Updated the PR description. I'm going to defer the proposed example/demo template for followup, as that will likely have it's own set of review feedback and want to decouple the concerns. |
arnarg
left a comment
There was a problem hiding this comment.
I have tested this locally and read through the code and I think this looks great. Love the de-duplication with the different backends.
Amazing work! ⭐
Adds CRD generators that return Nix values instead of building a generated
.nixfile, plus their chart and raw-object counterparts — completing the APImatrix.
Summary
fromCRDfromChartCRDfromCRDModulefromChartCRDModulecrdObjectscrdObjectsFromChartMotivation
fromCRDrenders the resource options to Nix source,nixfmts it, writes a.nixfile, and youimportit back — a serialize→deserialize round-trip plusan import-from-derivation. For flake/programmatic consumers that just want the
options, that file is pure overhead.
nixidy.applicationImportsalready acceptsfunctionTo attrs, sofromCRDModulereturns the module value directly anddrops straight in — no file, no import IFD.
crdObjects/crdObjectsFromChartexpose the rawCustomResourceDefinitionmanifests as values (e.g. to apply them to a cluster), deliberately
deployment-agnostic — they hand back objects and let the caller decide.
What's in here
fromCRDModule— native module value; no generated file.walk.nix, driving a text backend (source, for the committed-file workflow) and a value backend (live module).generator.nix/module.nixbecome thin assemblers over the one walk, so the branch logic lives in a single place.crdObjects— raw CRD manifests from asrc.fromChartCRDModule/crdObjectsFromChart, sharing onehelm template(viaklib.buildHelmChart);kubeVersionis now overridable across the chart family (incl.fromChartCRD), defaulting to nixidy's nixpkgs version — previously hardcoded.crdsargument is split intocrdFiles(source-based) andkindFilter(chart-based), since it meant a list of YAML files in one family and a kind filter in the other.crdsis kept as a deprecated alias on all six accessors (new arg wins; falling back warns and points at the new name).typed_resources.md(matrix-led, with the new accessors +kubeVersion),architecture.md(shared walk + backends),llms.txt.nix run .#crdAccessorTest).Backwards compatibility / verification
fromCRD/fromChartCRDfile output is unchanged — verifiednix build .#generators.argocdyields the same store path before/after the backend refactor.fromChartCRDonly gains an optionalkubeVersionwith its prior default, plus thecrds→kindFilteralias.moduleTests42/42,crd2jsonschemaTest18/18,crdAccessorTest(10 checks: chart↔src equivalence,kubeVersionhonored, deprecated-crds-alias resolution,specialMapKeyson the value backend),staticCheck.Done
llms.txtforfromCRDModule,crdObjects,fromChartCRDModule,crdObjectsFromChart, andkubeVersion.main(feat: add optional kindFilter to fromCRD #98 dropped from range).Follow-ups (separate PR)
fromCRDModuleviaapplicationImports(andcrdObjectsto deploy the CRDs). Deferred to its own PR: templates are inherently opinionated and warrant separate review.Notes for review
helpers exist both as source (inlined by the text backend, so committed
standalone files stay self-contained) and as values (
runtime.nix, for thenative modules). Live types/functions can't be serialized back to source, and
committed files can't reference nixidy internals — so the two forms must
coexist. The schema walk (the real divergence risk) is single-sourced.